LLM Ops

See:

• ML Ops
• LLMs

Resources

• Data-centric MLOps and LLMOps | Databricks
• LLMops 101: A Detailed Insight into Large Language Model Operations | Medium

Courses

• #COURSE LLMOps - DeepLearning.AI

Code

• #CODE GitHub - AgentOps-AI/agentops
  • Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen
  • AgentOps
  • Agent Tracking with AgentOps | AutoGen
  • AgentOps, the Best Tool for AutoGen Agent Observability | AutoGen
• #CODE Langsmith langchain-ai/langsmith-cookbook (github.com)
  • LangSmith (langchain.com)
  • Get started with LangSmith | 🦜️🛠️ LangSmith (langchain.com)
  • Evaluate your LLM application | 🦜️🛠️ LangSmith (langchain.com)

Serving LLMs, VLMs

• GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs
  • https://docs.vllm.ai/en/stable/index.html

References

• #PAPER Maximizing User Experience with LLMOps-Driven Personalized Recommendation Systems (2024)
• #PAPER LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs (2024)